[FEDE-8281] Upgrade Arrow Java to siren-18.3.0-1 by GeorgeAp · Pull Request #2 · sirensolutions/siren-arrow-java

GeorgeAp · 2026-01-06T10:49:04Z

What's Changed

Upgrade Arrow Java to 18.3..0

Change logs from Arrow 16.0.0 to Arrow Java 18.3.0
Arrow 16.0.0 - https://arrow.apache.org/blog/2024/04/20/16.0.0-release/
Arrow 17.0.0 - https://arrow.apache.org/blog/2024/07/16/17.0.0-release/
Arrow 18.0.0 - https://arrow.apache.org/blog/2024/10/28/18.0.0-release/
Arrow-Java 18.2.0 to 18.3.0 - https://github.com/apache/arrow-java/releases

Please fill in a description of the changes here.

This contains breaking changes.

Closes https://sirensolutions.atlassian.net/browse/FEDE-8281

GeorgeAp · 2026-01-08T09:00:33Z

memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java

-                    } catch (NoSuchMethodException e) {
-                      logger.debug("Cannot get constructor for direct buffer allocation", e);
-                      return e;
-                    } catch (SecurityException e) {


@torito this change was due to the [2021-11-24T15:22:37,545][DEBUG][s.i.n.u.i.PlatformDependent0] [mars] direct buffer constructor: unavailable reported in the PR sirensolutions/arrow#17.
The issue was widely discussed in google/gson#1875 and google/gson#1902

This is the exception thrown if we revert our change

java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.memory.RootAllocator at __randomizedtesting.SeedInfo.seed([9E683B735110AFF2:76856F24D3CDD0A6]:0) ... federate calls at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "SUITE-UTFHelperTest-seed#[DF6017972AC13922]-worker"] at org.apache.arrow.memory.unsafe.UnsafeAllocationManager.<clinit>(UnsafeAllocationManager.java:29) at org.apache.arrow.memory.unsafe.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:421) at java.base/java.lang.Class.forName(Class.java:412) at org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:105) at org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:92) at org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:826)

Those discussions on gson are not entirelly related, can you please:

instead of generic catch(Exception e), add a new catch (RealReceivedException e) {...} // <-- this to propose an upstream fix

Explain why if the exception is thrown, federate still works ? Direct Buffer is not needed for us ? what do we use instead ?

The real exception is not InaccessibleObjectException ?

It is actually that

java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,long) accessible: module java.base does not "opens java.nio" to unnamed module @2d6e8792

can we add only a new catch for InaccessibleObjectException instead of a generic one

Dont replace, add a new catch entry

I reverted our change and added it in-place of the SecurityException d6fca40, wdyt?

I have added the SecurityException back

torito · 2026-02-04T15:00:49Z

memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java


  /** The unsafe object from which to access the off-heap memory. */
-  private static final Unsafe UNSAFE;
+  public static final Unsafe UNSAFE;


why we need those to be public ?, where we are accessing it in Federate ?

We access it in Federate and keeping it private breaks our usage

In federate we are already accessing the Unsafe, see https://github.com/sirensolutions/siren-platform/blob/f32ca909badbf0527ffe90231fa5b52ee36a3d35/core/src/main/java/io/siren/federate/core/common/ReflectionHelper.java#L129
So, instead of geting the Unsafe from MemoryUtil, we access it by using the one we get from the ReflectionHelper ? This to limiting dependency on the fork, wdyt ?

Yes, good point, I will test this

I have reverted the change to keep it private in 003619b

torito · 2026-02-04T15:38:48Z

memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java


  /** The start offset of array data relative to the start address of the array object. */
-  private static final long BYTE_ARRAY_BASE_OFFSET;
+  public static final long BYTE_ARRAY_BASE_OFFSET;


The same for this offset, it seems we are also getting this value in Federate by using reflection as it is done here

…ce we have access to them in Federate through the ReflectionHelper

torito · 2026-02-05T09:02:30Z

memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java


      // get the offset of the address field in a java.nio.Buffer object
      Field addressField = java.nio.Buffer.class.getDeclaredField("address");
-      addressField.setAccessible(true);


This line should be removed also ? this code block is throwing an exception any way, maybe even before reaching this line, so maybe we dont need to remove it

I will validate this, the exception is currently thrown at L103 constructor.setAccessible(true);

…access and arrow.enable_unsafe_memory_access

Apply siren versioning and changes

9f82e76

GeorgeAp mentioned this pull request Jan 6, 2026

[FEDE-8281] Upgrade Arrow Java to 19.0.0 #1

Closed

GeorgeAp commented Jan 8, 2026

View reviewed changes

GeorgeAp marked this pull request as ready for review January 8, 2026 09:00

GeorgeAp requested review from scampi and torito February 3, 2026 14:48

torito reviewed Feb 4, 2026

View reviewed changes

GeorgeAp added 2 commits February 4, 2026 16:16

update generic exception to InaccessibleObjectException in MemoryUtil

d6fca40

add back the SecurityException

4e642e1

torito reviewed Feb 4, 2026

View reviewed changes

revert changes that make UNSAFE and BYTE_ARRAY_BASE_OFFSET public sin…

003619b

…ce we have access to them in Federate through the ReflectionHelper

torito reviewed Feb 5, 2026

View reviewed changes

GeorgeAp added 4 commits February 5, 2026 14:47

revert changes made to BoundsChecking for drill.enable_unsafe_memory_…

6238425

…access and arrow.enable_unsafe_memory_access

revert changes to MemoryUtil

e4dd5e5

we need to catch the InaccessibleObjectException in MemoryUtil

9d7964e

remove addressField.setAccessible(true) in MemoryUtil

1e514ee

Conversation

GeorgeAp commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's Changed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GeorgeAp Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torito Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GeorgeAp commented Jan 6, 2026 •

edited

Loading

GeorgeAp Feb 3, 2026 •

edited

Loading

torito Feb 4, 2026 •

edited

Loading